Author's response to reviews Title: The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research Authors:

“more conceptually-driven research” – not clear what this means...I would argue that this paper is weak on the conceptual stuff • We have clarified in the abstract that future research should be informed by conceptualizations and models of sustainability. We note on page 6 that our goal for this project was to examine how sustainability has been studied thus far and to summarize the conclusions that have been drawn to date to the best of our ability given the state of the current literature. Thus, a full consideration of existing models and conceptualizations of sustainability is beyond the scope of the current paper. However, we do appreciate both Reviewer 2 and 3’s suggestions that we present more of a discussion of sustainability and its study and that the discussion include our own views and conclusions. We have done so in the discussion section of the paper, and our ongoing review of the conceptual literature has informed this discussion. “Sustainability must be treated as a distinct outcome” – I would say this kind of language is part of the problem, not part of the solution. I’d suggest that sustainability sits in tension with adaptation and embedding, and therein lies the key research agenda!...The whole of this introduction suggests that the optimal achievement is for an intervention to be maintained without adaptation. This is not what the background literature suggests is necessarily the case. I challenge these authors to defend themselves against the charge of ‘logic model bias’. • As mentioned in our response to Reviewer 1, we now discuss the differing views on sustainability on page 5. While in our original submission, we acknowledged the importance of adaptation (and the need to understand how it impacts sustainability and the program or intervention’s desired outcomes c.f., page 20-22), we agree that the introduction was perhaps shaped primarily by the way we think about health care interventions that require implementation with fidelity. In addition to the changes noted in our second response to Reviewer 1’s discretionary revisions, we have revised our introduction and discussion sections to reflect a broader perspective that takes the implementation of other types of programs and interventions into account. Top of page 8 – referring to ‘factors’ and suggesting that each of these factors needs looking at. My reading of the wider sustainability literature is that it’s moving from a rather static focus on ‘factors’ (which was certainly where the interest was a few years back) towards a much more dynamic study of how different influences interact with one another and change over time, and also towards studying the things which are difficult to report (such as local micropolitics)..... The findings seem to skirt round all the things we know in our bones are important in sustainability, such as relationships, energy, commitment, goodwill, values etc. I wonder why that was – must be an artefact of the research approach in both the primary studies in their dataset and then in their method of analysis. • Our goal for this review was to survey the empirical literature to determine how sustainability has been studied thus far. Our findings indicate that to date in the empirical literature, relatively little explicit consideration has been given to interactions between factors associated with sustainability. Many of the studies that we reviewed were conducted several years ago when the focus on factors was more prevalent. Thus, a number of them did present their findings in terms of factors and characteristics—although others did highlight “processes” that can be viewed as interactions between different influences and/or some of the things mentioned in this review (e.g., collaborative relationships/partnerships). We agree that authors of some more recent articles acknowledge the importance of studying the ways in which influences interact and we look forward to seeing more research that sheds light on these phenomenon. (It is worth noting, however, that in a new paper Scheirer and Dearing also discuss some disadvantages of a “process” definition of sustainability). We have shifted our discussion from a consideration of “factors” (which were identified in a number of the quantitative studies we reviewed) to a discussion of influences, which we believe is a more appropriate term. We have augmented the discussion from our previous submission on the need to understand the way that these influences interact and shape interventions and programs over time (c.f., p. 23-24). We also acknowledge some of the limitations of the research methods employed to date in terms of its ability to capture the processes and influences that the reviewer describes above on page 23-24. Page 10 – literature review search strategy seems to have been good for identifying empirical studies on sustainability and related concepts, but they didn’t describe how they identified the theoretical literature in the background section. • The focus was primarily on the empirical studies because for this paper we wanted to examine how sustainability has been studied in the empirical literature. However, for this and other work that we have in process, we have been using similar methods to identify theoretical literature. We appreciated the reviewer’s comments as they prompted us to complete another search for conceptual papers and reviews, which we have discussed above. Page 13 – an old bugbear of mine: high inter-rater agreement doesn’t mean the codings are correct, merely that both ‘independent’ assessors approached the literature with the same assumptions. But again, I recognise that this ‘objective’ approach is absolutely de rigeur for Imp Science so is ‘in paradigm’. • We have updated and retained our report of inter-rater agreement, given that it is ‘in paradigm’ for systematic reviews. Page 13 – please don’t express proportions in percentages when the denominator is less than 100. It conveys a spurious level of accuracy and is bad Latin. So 5% of 61 is how many studies? Easier to say three, surely? • Because we expanded our search to include 2011 papers, and also included the papers that were found in the review mentioned above, we now have over 100 papers. We report on percentages of the full sample when appropriate but report numbers of studies when our denominator is below 100. Page 15 “Only ten studies reported on the quality or level of fidelity of implementation, and only eight of these used a form of independent observation and assessment to evaluate sustainability outcomes.” This seems to me to be the nugget in this paper. ... I would reflect on this finding, dig out more detail on it, and expand discussion on it. • We appreciate this suggestion, as we do believe that this is an important finding to highlight. We have discussed fidelity in greater detail on pages 20-22 in the context of a discussion on when fidelity vs. adaptation is appropriate or necessary. Page 17 “Several of our findings were surprising. First, given the emphasis on implementation in the fields of health care and mental health, relatively few studies in these areas examined sustainability.” It reads as odd that they found it surprising since they begin by citing my own team from 2004 observing that hardly anyone studies sustainability! • In part because your seminal review indicated a dearth of research in this area, we would have expected to see a greater consideration of sustainability in the literature by now. However, given the large number of papers published in the first half of 2011 alone, it appears that the study of sustainability is in fact increasing. We have thus removed our expression of surprise. An arguable omission in the whole paper – the link between sustainability of a programme and sustainability as in the ‘green agenda’. See Dave Pencheon’s work on this. He links the two. • We appreciate this suggestion and have reviewed Dr. Pencheon’s work. However, we believe it is important to to retain the focus of the current paper primarily on the empirical literature and recommendations for further study. We will consider his work as we review existing conceptualizations of sustainability.